Light sensor chip and image processing device adaptable to low illumination environment

ABSTRACT

There is provided an image processing device including a light sensor and a processor. The light sensor is used to detect light and output an image frame. The processor identifies intensity of ambient light according to an image parameter associated with the image frame. When the ambient light is identified to be strong enough, the processor performs an object identification directly using the image frame. When the ambient light is identified to be not enough, the processor firstly converts the image frame to a converted image using a machine learning model, and then performs the object identification using the converted image.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. Ser. No.16/257,277, filed on Jan. 25, 2019, the disclosure of which is herebyincorporated by reference herein in its entirety.

BACKGROUND 1. Field of the Disclosure

This disclosure generally relates to an image processing technique and,more particularly, to an image processing device adaptable to lowillumination environment and an operating method thereof.

2. Description of the Related Art

The conventional image capturing device generally has an adjustableexposure time. In a strong light environment, the image capturing deviceacquires an image using a short exposure time. In a weak lightenvironment, the image capturing device acquires an image using a longexposure time so as to increase image features in the captured imagesuch that the object identification can be performed correctly.

However, when this kind of image capturing device is applied to acquirean object image of a moving object in the weak light environment,problems can occur. Because the exposure time is significantly extended,the relative displacement between the moving object and the imagecapturing device within the extended exposure time can reflect in theacquired image to have blurred object image. More significantly, if thisimage capturing device is operated in a fast moving scenario, an objectimage may not even be captured in some images.

Accordingly, it is necessary to provide an image processing device thatcan acquire valid images even in a weak light environment so as toimprove the identification accuracy.

SUMMARY

The present disclosure provides a light sensor chip, an image processingdevice and an operating method thereof that do not adjust an exposuretime of a light sensor from a strong light environment to a weak lightenvironment.

The present disclosure further provides a light sensor chip, an imageprocessing device and an operating method thereof that convert a shortexposure image to a quasi-long exposure image using a machine learningmodel to increase image features in the captured image thereby improvingthe operating accuracy.

The present disclosure provides a light sensor chip including a lightsensor and a processor. The light sensor is configured to detect lightusing a first exposure time to output a first image. The processor iselectrically connected to the light sensor to receive the first image.When identifying that a gain value for amplifying the first image issmaller than a gain threshold, the processor is configured to output thefirst image. When identifying that the gain value for amplifying thefirst image is not smaller than the gain threshold, the processor isconfigured to convert the first image into a converted image using apre-stored learning model, and then output the converted image.

The present disclosure further provides an image processing deviceincluding a light sensor chip and an electronic device. The light sensorchip is configured to detect light using a first exposure time to outputa first image. The electronic device is coupled to the light sensorchip, and includes a processor. When identifying that a gain value foramplifying the first image is smaller than a gain threshold, theprocessor is configured to perform an object identification using thefirst image. When identifying that the gain value for amplifying thefirst image is not smaller than the gain threshold, the processor isconfigured to convert the first image into a converted image using apre-stored learning model, and then perform the object identificationusing the converted image.

The present disclosure further provides a light sensor chip including alight sensor and a processor. The light sensor is configured to detectlight using a first exposure time to output a first image. The processoris electrically connected to the light sensor to receive the firstimage. When identifying that a convergence time of auto exposure issmaller than a time threshold, the processor is configured to output thefirst image. When identifying that the convergence time of auto exposureis not smaller than the time threshold, the processor is configured toconvert the first image into a converted image using a pre-storedlearning model, and then output the converted image.

In the embodiments of the present disclosure, when the brightness of anoutput image from a light sensor is identified to be not enough (e.g.,according to image gray levels, image quality and/or according to thegain value, exposure time and convergence time of auto exposure of thelight sensor), the output image is not directly post-processed butfirstly converted to a quasi-long exposure image that contains moreimage features, and then the quasi-long exposure image ispost-processed.

In the embodiments of the present disclosure, the object identificationrefers to the gesture recognition, distance identification, speedidentification, face recognition and depth map construction.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects, advantages, and novel features of the present disclosurewill become more apparent from the following detailed description whentaken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram of an image processing device according to afirst embodiment of the present disclosure.

FIG. 2 is a block diagram of an image processing device according to asecond embodiment of the present disclosure.

FIG. 3 is a flow chart of an operating method of an image processingdevice according to one embodiment of the present disclosure.

FIG. 4 is a schematic diagram of the exposure times of an imageprocessing device according to some embodiments of the presentdisclosure.

FIG. 5 is an operational schematic diagram of an image processing deviceaccording to one embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT

It should be noted that, wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The light sensor chip, image processing device and an operating methodof the present disclosure are adaptable to an electronic device that hasa relative displacement with respect to an object, such as a cleaningrobot, a self-driving car, a drone, a gesture recognition system, a facerecognition device, but not limited to. By keeping the same exposuretime or slightly extending the exposure time in a low light environment,the problem of acquiring an invalid image that causes operation error issolved, wherein said invalid image is referred to an image framecontaining a blurred object image or missing an object image.

Please referring to FIG. 1, it is a block diagram of an image processingdevice 100 according to a first embodiment of the present disclosure.The image processing device 100 includes a light sensor chip 11 and anelectronic device 13. In one aspect, the light sensor chip 11 isincluded in the electronic device 13, e.g., the light sensor chip 11being arranged on a casing of the electronic device 13 to acquire imagesof external environment. In another aspect, the light sensor chip 11 iscoupled to the electronic device 13 via a communication interface towirelessly transmit an output image to the electronic device 13. Asdescribed below, the output image is a raw data image, a pre-processedimage or a converted image.

The light sensor chip 11 includes a light sensor 111 and a processor (orreferred to chip processor) 113. For example, the light sensor 111 andthe processor 113 are arranged in the same encapsulation.

The light sensor 111 is, for example, a CCD image sensor, a CMOS imagesensor or the like. The light sensor 111 includes a plurality of pixelsarranged in a matrix. The light sensor 111 is used to detect, using afixed or adjustable exposure time, light within a field of view FOV tooutput an image frame, e.g., a first image Im_raw. The light sensor 111preferably has multiple amplifiers (e.g., programmable gain amplifier,PGA) for amplifying raw data outputted by the pixel array using a gainvalue. The light sensor 111 preferably has auto exposure mechanism.

The processor 113 is, for example, a digital signal processor (DSP) orapplication specific integrated circuit (ASIC). The processor 113 iselectrically connected to the light sensor 111 to receive a first imageIm_array, which is a raw data image or an amplified raw data image. Theprocessor 113 identifies an operating mode (or ambient light intensity)according to an image parameter associated with the first image Im_raw,wherein the image parameter includes at least one of the imagebrightness, gain value, convergence time of auto exposure and imagequality. The image parameter is calculated by the processor 113 usingthe first image Im_raw.

The processor 113 further includes a memory for storing thresholdsassociated with the above image parameter to be compared with a currentimage parameter, which is calculated by the processor 113 according to acurrent first image. The processor 113 identifies the intensity ofambient light according to a comparison result of comparing the currentimage parameter and the stored threshold to determine an operating mode.For example, when an average brightness value of the first image Im_rawis larger than a brightness threshold, when a gain value (analog gainand/or digital gain) for amplifying the first image Im_raw is smallerthan a gain threshold, when an image quality of the first image Im_rawis larger than a quality threshold and/or a convergence time of autoexposure is smaller than a time threshold, the processor 113 identifiesthat the ambient light is strong enough and thus a strong light mode isentered; otherwise, a weak light mode is entered.

In the present disclosure, the processor 113 is further embedded orpre-stored with a learning model 115 that is implemented by a hardwarecircuit and/or software codes. The learning model 115 is generated,before the shipment of the image processing device 100, by a computerdevice and/or web platform running a data network architecture (e.g.,including neural network learning algorithm, deep learning algorithm,but not limited to). The data network architecture uses a raw data imageacquired by the light sensor 111 with the first exposure time to learnto generate a ground truth image, and generates the learning model 115to be recorded in the image processing device 100. The learning modelmay be different corresponding to different light sensors.

For example referring to FIG. 5, the first exposure time is referred to,for example, an exposure time used by the light sensor 111 under astrong light environment, e.g., the first exposure time being referredto a short exposure. Accordingly, if this short exposure is also used ina weak light environment, the obtained raw data image has a poor imagefeature, e.g., the raw data image shown in FIG. 5.

In addition, for generating the machine learning model, before theshipment of the image processing device 100, the light sensor 111acquires a ground truth image (not containing any image of a movingobject) in a weak light environment using an exposure time, referred toa long exposure, longer than the first exposure time. As the longexposure is longer than the short exposure, the ground truth image has abetter image feature, as shown in FIG. 5, than the raw data image. Thelong exposure is preferably selected as a time interval to cause theground truth image to have an image feature higher than a featurethreshold, e.g., selecting a maximum usable exposure time of the lightsensor 111. More specifically, the image feature contained in the groundtruth image is enough for performing the object identificationcorrectly. The image feature includes, for example, the image quality,contrast, clarity or the like.

As mentioned above, after the light sensor 111 outputs the first imageIm_raw, the processor 113 identifies an operating mode according to theassociated image parameter. When identifying that the operating mode isa strong light mode, the processor 113 directly outputs the first imageIm_raw to the electronic device 13. The processor 133 (or referred todevice processor such as a microcontroller unit, a central processingunit, a graphic processing unit or the like) of the electronic device 13performs the post-processing, e.g., identifying the gesture, distance,moving speed, moving direction, face and so on, according to at leastone first image Im_raw. In some embodiments, the processor 113 performsthe pre-processing such as the filtering, denoising and/or digitizing onthe first image Im_raw to generate a pre-processed image Im_pre.

On the other hand, when identifying that the operating mode is a weaklight mode, the processor 113 converts, using the pre-stored learningmodel 115, the first image Im_raw to a converted image Im_adj, and thenoutputs the converted image Im_adj to the electronic device 13. Forexample referring to FIG. 5, although the converted image Im_adj isshown not to have such a good image feature as the ground truth image,the converted image Im_adj has a higher image quality, contrast andclarity and has a lower blurring than the first image Im_raw such thatthe converted image Im_adj is suitable for the object identification.

More specifically, in the first embodiment, when the ambient light isstrong enough (i.e. strong light mode), the light sensor chip 11directly outputs a first image Im_raw to the electronic device 13 forthe post-processing. Because the first image Im_raw captured in thiscondition has enough image features, the first image Im_raw is notconverted by the learning model 115. However, when the ambient light isnot enough (i.e. weak light mode), the light sensor chip 11 does notoutput the first image Im_raw but outputs a converted image Im_adj whichis processed by the learning model 115. Accordingly, it is able toacquire valid images (not containing blurred object image) withoutextending the exposure time, and a high frame rate is maintained underweak illumination.

Referring to FIG. 4, in the first embodiment, the processor 113 furtherselects to control the light sensor 111 to detect light within the FOVusing a second exposure time ET2 to output a second image in the weaklight mode, wherein the second exposure time ET2 is longer than thefirst exposure time ET1. To avoid the problems in the traditional imagecapturing device, when a blurring of the second image is higher than apredetermined blurring threshold, the processor 113 controls the lightsensor 111 to shorten the second exposure time to prevent from acquiringblurred images. Similarly, in the weak light mode, the processor 113uses the pre-stored learning model 115 to convert the second image intoanother converted image, and then outputs said another converted image.The processor 133 of the electronic device 13 performs the objectidentification according to at least one said another converted image.

For example, when identifying that the image feature of the convertedimage Im_adj is lower than a predetermined feature threshold, theprocessor 133 informs the light sensor chip 11 to acquire image framesusing a longer exposure time, e.g., the second exposure time ET2.

Please referring to FIG. 2, it is a block diagram of an image processingdevice 200 according to a second embodiment of the present disclosure.The difference between the image processing device 200 and the imageprocessing device 100 of the first embodiment is that the learning model235 is arranged in the electronic device 23 instead of in the lightsensor chip 21, but other parts are similar to the first embodiment.

The image processing device 200 includes a light sensor chip 21 and anelectronic device 23. The light sensor chip 21 may also be arranged inor outside the electronic device 23 according to different applications.In the aspect that the light sensor chip 21 is arranged in theelectronic device 23, the learning model 235 is executed by an externalprocessor (e.g., processor 233) of the light sensor chip 21.

The light sensor chip 21 also includes a light sensor 211 (identical tothe light sensor 111) and a processor 213 (identical to the processor113), only the processor 213 does not have the learning model 235. Thelight sensor 211 of the light sensor chip 21 is also used to detectlight within a field of view FOV using a first exposure time to output afirst image Im_raw. According to different applications, the processor213 directly outputs the first image Im_raw, or performs apre-processing (e.g., filtering, denoising and/or digitizing) to outputa pre-processed image Im_pre to the electronic device 23.

The electronic device 23 is coupled (wired or wirelessly) to the lightsensor chip 21 to receive the first image Im_raw or the pre-processedimage Im_pre. The electronic device 233 further includes a processor 233(or referred to device processor such as CPU, MCU or GPU) used toidentify an operating mode according to an image parameter associatedwith the first image Im_raw. As mentioned above, the image parameterincludes at least one of the image brightness, gain value, convergencetime of auto exposure and image quality. The method of identifying anoperating mode has been illustrated in the first embodiment, and thusdetails thereof are not repeated herein.

When identifying that the operating mode is a strong light mode, theprocessor 233 uses the first image Im_raw (or pre-processed imageIm_pre) to perform an object identification. When identifying that theoperating mode is a weak light mode, the processor 233 converts, usingembedded or pre-stored learning model 235, the first image Im_raw (orpre-processed image Im_pre) into a converted image Im_adj at first, andthen use the converted image Im_adj to perform the objectidentification. It should be mentioned that although FIG. 2 shows thatthe learning model 235 is outside of the processor 233, it is onlyintended to illustrate but not to limit the present disclosure. Thelearning model 235 is included in the processor 233 and implemented by ahardware circuit and/or software codes.

In the second embodiment, the image quality, contrast and clarity of theconverted image Im_adj are higher than those of the first image Im_raw,or the blurring of the converted image Im_adj is lower than that of thefirst image Im_raw such that the object identification is performedcorrectly in a low light environment and using a short exposure time,and this effect cannot be achieved using conventional image capturingdevices.

In the second embodiment, the learning model 235 is generated, beforethe shipment of the image processing device 200, by a computer deviceand/or web platform running data network architecture. The data networkarchitecture uses a raw data image acquired by the light sensor chip 21with the first exposure time to learn to generate a ground truth image,and generates the learning model 235 to be recorded in the imageprocessing device 200. As mentioned above, the ground truth image isacquired by the light sensor chip 21 using an exposure time longer thanthe first exposure time (e.g. a selectable longest exposure time of thelight sensor 211), and has image features higher than a featurethreshold, for example referring to FIG. 5.

More specifically, in the second embodiment, the light sensor chip 21 isused to output a raw data image (i.e. the first image Im_raw) or apre-processed raw data image (i.e. the pre-processed image Im_pre).After identifying the intensity of ambient light, the electronic device23 then determines whether to perform an object identifying according tothe raw data image, or to convert the raw data image to a convertedimage at first and then perform the object identifying according to theconverted image. That is, in the weak light mode the processor 233 doesnot use the first image Im_raw to perform the object identification.

Similarly, the processor 233 of the electronic device 23 furthercontrols the light sensor chip 21 to detect light using a secondexposure time (e.g., while the first image or converted image havingpoor image feature) and output a second image, referring to FIG. 4,wherein the second exposure time ET2 is longer than the first exposuretime ET1. The processor 233 uses the pre-stored learning model 235 toconvert the second image into another converted image, and use saidanother converted image to perform the object identification.

The processor 233 further adjusts the second exposure time according toa comparison result of comparing the blurring of the second image with ablurring threshold, e.g., shortening the second exposure time while theblurring of the second image is higher than the blurring threshold.

Referring to FIG. 3, it is a flow chart of an operating method of animage processing device according to one embodiment of the presentdisclosure, wherein the method is adaptable to the image processingdevice of the first embodiment in FIG. 1 and the second embodiment inFIG. 2. Details of this operating method are illustrated below using anexample.

Firstly, a light sensor (111 or 211) uses a first exposure time todetect light and output a first image Im_raw. In one aspect, the firstexposure time is an exposure time used under strong light environment,e.g., the shortest exposure time of the light sensor. In another aspect,the light sensor has only one fixed exposure time.

Next, a processor (e.g., 113 or 233) compares an image parameterassociated with the first image Im_raw with a parameter threshold, StepS31. As mentioned above, the image parameter is a proper parameter foridentifying the intensity of ambient light, e.g., including at least oneof the image brightness, gain value, convergence time of auto exposureand image quality. The image parameter is previously stored in a memoryof the device.

When the image parameter exceeds the parameter threshold, a strong lightmode is entered, Step S32. For example, if the image brightness andimage quality are larger, it means that the ambient light is stronger;whereas, if the gain value and convergence time of auto exposure aresmaller, it means that the ambient light is stronger, and a properthreshold is selected accordingly. That is, the exceeding is referred tothe image brightness or the image quality being larger than anassociated threshold, and the gain value or the convergence time beingsmaller than an associated threshold. Meanwhile, the processor (133 or233) directly uses the first image Im_raw to perform the objectidentification, e.g., including the object tracking, depthidentification, face recognition and so on.

When the image parameter does not exceed the parameter threshold, a weaklight mode is entered, e.g., Step S33. Meanwhile, the processor (133 or233) uses a pre-stored learning model (115 or 235) to convert the firstimage Im_raw into a converted image Im_adj, and then uses the convertedimage Im_adj to perform the post-processing such as the objectidentification, Step S34.

The method of generating the learning model has been illustrated above,e.g., referring to FIG. 5, and thus details thereof are not repeatedherein. As the converted image Im_adj is generated for improving theidentification accuracy, the image quality, contrast or clarity of theconverted image Im_adj is higher than that of the first image Im_raw, orthe blurring of the converted image Im_adj is lower than that of thefirst image Im_raw. More specifically speaking, in this aspect, thefirst image Im_raw is considered not having enough image features in theweak light environment, and thus the first image Im_raw is not used toperform the object identification.

However, when the image feature of the first image Im_raw is too low(e.g., lower than a predetermined threshold), the light sensor (111 or211) is further controlled to detect light using a second exposure timeto output a second image, wherein the second exposure time is longerthan the first exposure time, Step S331-S332. The processor (113 or 233)uses the pre-stored learning model (115 or 235) to convert the secondimage into another converted image, and then uses said another convertedimage to perform the object identification. It should be mentioned that,steps S331-S332 are not necessary to be implemented.

The effect of the present disclosure is illustrated below. When a fixedexposure time is used, the frame rate of image frames outputted by alight sensor is not decreased under low illumination such that correctoperation is performed even in the high relative speed scenario. Whenthe present disclosure is applied to a gesture recognition device, thegesture recognition device is arranged to be connected to a display tocontrol a cursor movement thereon in some applications. In the low lightenvironment, as the exposure time is not extended significantly, thecursor trace shown on the display is not broken (due to losing objectimage) even in the high relative speed scenario. When the presentdisclosure is applied to a face recognition device, the face recognitiondevice is arranged to output a trigger signal to indicate a face imagebeing recognized in some applications. In the low light environment, asthe exposure time is not extended significantly, even though the targetto be recognized is moving, a condition that the trigger signal unableto be generated does not occur.

As mentioned above, the conventional image capturing device extends theexposure time in a low light environment to capture enough imagefeatures for the object identification. However, in detecting a movingobject, the long exposure time can cause another problem such as objectimage blurring or unable to capture any object image. Accordingly, thepresent disclosure provides an image processing device (e.g., FIGS. 1 to2) and an operating method thereof (e.g., FIG. 3) that maintain the sameexposure time even in a low light environment to avoid the problem ofacquiring invalid images. Meanwhile, an image acquired using a shortexposure time is converted to increase the image feature using a machinelearning algorithm to achieve the effect of accurately performing theobject identification event in the low light environment.

Although the disclosure has been explained in relation to its preferredembodiment, it is not used to limit the disclosure. It is to beunderstood that many other possible modifications and variations can bemade by those skilled in the art without departing from the spirit andscope of the disclosure as hereinafter claimed.

What is claimed is:
 1. A light sensor chip, comprising: a light sensorconfigured to detect light using a first exposure time to output a firstimage; and a processor electrically connected to the light sensor toreceive the first image, wherein when identifying that a gain value foramplifying the first image is smaller than a gain threshold, theprocessor is configured to output the first image, and when identifyingthat the gain value for amplifying the first image is not smaller thanthe gain threshold, the processor is configured to convert the firstimage into a converted image using a pre-stored learning model, and thenoutput the converted image.
 2. The light sensor chip as claimed in claim1, wherein the learning model is trained by a data network architecturebased on a ground truth image.
 3. The light sensor chip as claimed inclaim 1, wherein an image quality, a contrast or a clarity of theconverted image is higher than that of the first image, or a blurring ofthe converted image is lower than that of the first image.
 4. The lightsensor chip as claimed in claim 1, wherein the processor does not outputthe first image when identifying that the gain value for amplifying thefirst image is not smaller than the gain threshold.
 5. The light sensorchip as claimed in claim 1, wherein when identifying that the gain valuefor amplifying the first image is not smaller than the gain threshold,the processor is further configured to control the light sensor todetect light using a second exposure time to output a second image,wherein the second exposure time is longer than the first exposure time,and convert, using the pre-stored learning model, the second image intoanother converted image, and then output the another converted image. 6.The light sensor chip as claimed in claim 5, wherein when a blurring ofthe second image is higher than a blur threshold, the processor isfurther configured to shorten the second exposure time.
 7. An imageprocessing device, comprising: a light sensor chip configured to detectlight using a first exposure time to output a first image; and anelectronic device coupled to the light sensor chip, and comprising aprocessor, wherein when identifying that a gain value for amplifying thefirst image is smaller than a gain threshold, the processor isconfigured to perform an object identification using the first image,and when identifying that the gain value for amplifying the first imageis not smaller than the gain threshold, the processor is configured toconvert the first image into a converted image using a pre-storedlearning model, and then perform the object identification using theconverted image.
 8. The image processing device as claimed in claim 7,wherein the learning model is trained by a data network architecturebased on a ground truth image.
 9. The image processing device as claimedin claim 7, wherein an image quality, a contrast or a clarity of theconverted image is higher than that of the first image, or a blurring ofthe converted image is lower than that of the first image.
 10. The imageprocessing device as claimed in claim 7, wherein the processor does notuse the first image to perform the object identification whenidentifying that the gain value for amplifying the first image is notsmaller than the gain threshold.
 11. The image processing device asclaimed in claim 7, wherein when identifying that the gain value foramplifying the first image is not smaller than the gain threshold, theprocessor is further configured to control the light sensor chip todetect light using a second exposure time to output a second image,wherein the second exposure time is longer than the first exposure time,and convert, using the pre-stored learning model, the second image intoanother converted image, and then perform the object identificationusing the another converted image.
 12. The image processing device asclaimed in claim 11, wherein when a blurring of the second image ishigher than a blur threshold, the processor is further configured toshorten the second exposure time.
 13. A light sensor chip, comprising: alight sensor configured to detect light using a first exposure time tooutput a first image; and a processor electrically connected to thelight sensor to receive the first image, wherein when identifying that aconvergence time of auto exposure is smaller than a time threshold, theprocessor is configured to output the first image, and when identifyingthat the convergence time of auto exposure is not smaller than the timethreshold, the processor is configured to convert the first image into aconverted image using a pre-stored learning model, and then output theconverted image.
 14. The light sensor chip as claimed in claim 13,wherein the learning model is trained by a data network architecturebased on a ground truth image.
 15. The light sensor chip as claimed inclaim 13, wherein an image quality, a contrast or a clarity of theconverted image is higher than that of the first image, or a blurring ofthe converted image is lower than that of the first image.
 16. The lightsensor chip as claimed in claim 13, wherein the processor does notoutput the first image when identifying that the convergence time ofauto exposure is not smaller than the time threshold.
 17. The lightsensor chip as claimed in claim 13, wherein when identifying that theconvergence time of auto exposure is not smaller than the timethreshold, the processor is further configured to control the lightsensor to detect light using a second exposure time to output a secondimage, wherein the second exposure time is longer than the firstexposure time, and convert, using the pre-stored learning model, thesecond image into another converted image, and then output the anotherconverted image.
 18. The light sensor chip as claimed in claim 17,wherein when a blurring of the second image is higher than a blurthreshold, the processor is further configured to shorten the secondexposure time.